Nonparametric Error Estimation Methods for Evaluating and Validating Artificial Neural Network Prediction Models

Typically the true error of ANN prediction model is estimated by testing the trained network on new data not used in model construction. Four well-studied statistical error estimation methods: cross-validation, group cross-validation, jackknife and bootstrap are reviewed and are presented as competing error estimation methodologies that could be used to evaluate and validate ANN prediction models. All four methods utilize the entire sample for the construction of the prediction model and estimate the true error via a resampling methodology. INTRODUCTION The evaluation and validation of an artificial neural network (ANN) prediction model is based upon some error function; usually mean squared error, mean absolute error, or if appropriate, percent incorrect classification. Since the objective of a prediction model is to predict successfully on new data, the true error of a model is statistically defined on "an asymptotically large number of new data points that converge in the limit to the actual population distribution" []. In most real world applications unlimited sample sizes are impossible or too costly to obtain. As a consequence the network modeler is faced with the "performance/evaluation" dilemma; on one hand, much data is needed to build or train the network model but on the other hand much data is also needed to get an accurate evaluation of the model. Most modelers opt for achieving a better performing model at the expense of a good evaluation. The focus of our current research seeks to answer the question; Can the true error of an ANN prediction model be empirically extrapolated from limited sample sizes? Typically the true error of an ANN prediction model is estimated by testing the trained network on new data not used in model construction. In cases where data is severely limited this procedure is not always performed. Consequently, the true error is estimated using the same data that was used to construct the model. This paper reviews four well-studied statistical error estimation methods: crossvalidation, group cross-validation, jackknife and bootstrap. All four methods utilize the entire sample for the construction of the prediction model and estimate the true error via a resampling methodology. These methods are currently used to evaluate and validate statistical prediction models. They have been shown to provide good error estimates, but can be computationally very expensive in terms of the number of prediction models constructed. The purpose of this paper is to present them as competing error estimation methodologies that could be used to evaluate and validate ANN prediction models. Some experimental results are given. THE PREDICTION PROBLEM Consider a class of prediction models, f (statistical or ANN), so that for any given xk we can predict yk. Assume that there is some unknown distribution F, from which a random sample, T (training set), is drawn: T={(x1, y1), (x2, y2), ... , (xn, yn)}; where ti=(xi,yi) and ti ~ iid F . T, of sample size n, is selected from a population of size N; where n<<N. Construct a model, $ f (T, x). $ f (T, x) is then used to predict yk from xk, where tk= (xk ,yk ) (test set) is not used in the construction of the prediction rule f (T, x). How good is $ f T,xk ε φ for k [1, N] ∈ ? The measure of the prediction error is determined according to some specified loss function L. The squared error will be used throughout this paper. The true error, Err, is the expected value over the entire population: Err = 1 N yi f T,xi i =1 N 2